Context Sensitive Overlays In Voice Controlled Headset Computer Displays

ABSTRACT

In headset computers that leverage voice commands, often the user does not know what voice commands are available. In one embodiment, a method includes providing a user interface in a headset computer and, in response to user utterance of a cue toggle command, displaying at least one cue in the user interface. Each cue can correspond to a voice command associated with code to execute. In response to user utterance of the voice command, the method can also include executing the code associated with the voice command. The user can therefore ascertain what voice commands are available.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/799,790 filed Mar. 13, 2013 (Attorney docket no. 0717.2116-005),which claims the benefit of U.S. Application No. 61/749,240 filed Jan.4, 2013 (Attorney docket no. 0717.2116-003) and is acontinuation-in-part of U.S. application Ser. No. 13/234,916 filed Sep.16, 2011, now U.S. Pat. No. 9,122,307 (Attorney docket no.0717.2116-001), which claims the benefit of 61/384,586 filed Sep. 20,2010 (Attorney docket no. 0717.2116-000). U.S. application Ser. No.13/799,790 filed Mar. 13, 2013, now U.S. Pat. No. 9,377,862 (Attorneydocket no. 0717.2116-005) also claims priority to and is acontinuation-in-part of U.S. application Ser. No. 13/799,888, filed Mar.13, 2013, now U.S. Pat. No. 9,377,862 (Attorney docket no.0717.2173-001) which claims the benefit of U.S. Application No.61/653,127, filed May 30, 2012 (Attorney docket no. 0717.2173-000). Theentire teachings of the above applications are incorporated herein byreference.

BACKGROUND OF THE INVENTION

Mobile computing devices, such as notebook PCs, smart phones, and tabletcomputing devices, are now common tools used for producing, analyzing,communicating, and consuming data in both business and personal life.Consumers continue to embrace a mobile digital lifestyle as the ease ofaccess to digital information increases with high-speed wirelesscommunications technologies becoming ubiquitous. Popular uses of mobilecomputing devices include displaying large amounts of high-resolutioncomputer graphics information and video content, often wirelesslystreamed to the device. While these devices typically include a displayscreen, the preferred visual experience of a high-resolution, largeformat display cannot be easily replicated in such mobile devicesbecause the physical size of such device is limited to promote mobility.Another drawback of the aforementioned device types is that the userinterface is hands-dependent, typically requiring a user to enter dataor make selections using a keyboard (physical or virtual) ortouch-screen display. As a result, consumers are now seeking ahands-free, high-quality, portable, color display solution to augment orreplace their hands-dependent mobile devices.

SUMMARY OF THE INVENTION

Recently developed micro-displays can provide large-format,high-resolution color pictures and streaming video in a very small formfactor. One application for such displays can be integrated into awireless headset computer worn on the head of the user with a displaywithin the field of view of the user, similar in format to eithereyeglasses, audio headset or video eyewear. A “wireless computingheadset” device includes one or more small high-resolutionmicro-displays and optics to magnify the image. The WVGA microdisplayscan provide super video graphics array (SVGA) (800×600) resolution orextended graphic arrays (XGA) (1024×768) or even higher resolutions. Awireless computing headset contains one or more wireless computing andcommunication interfaces, enabling data and streaming video capability,and provides greater convenience and mobility through hands dependentdevices. For more information concerning such devices, see co-pendingpatent applications entitled “Mobile Wireless Display Software Platformfor Controlling Other Systems and Devices,” U.S. application Ser. No.12/348,648 filed Jan. 5, 2009, “Handheld Wireless Display Devices HavingHigh Resolution Display Suitable For Use as a Mobile Internet Device,”PCT International Application No. PCT/US09/38601 filed Mar. 27, 2009,and “Improved Headset Computer,” U.S. Application No. 61/638,419 filedApr. 25, 2012, each of which are incorporated herein by reference intheir entirety.

In one embodiment, a method includes providing a user interface in aheadset computer and, in response to user utterance of a cue togglecommand, displaying at least one cue in the user interface. Each cue cancorrespond to a voice command associated with code to execute. Inresponse to user utterance of the voice command, the method can alsoinclude executing the code associated with the voice command. In anotherembodiment, the method can further includes displaying the interfacewithout the cue at least one of prior to the cue toggle command andafter a subsequent cue toggle command. Displaying the cue can includedisplaying words that activate the voice command. Displaying the cue canalso include displaying the cue in the user interface corresponding tothe voice command associated with the control, where the control isdisplayed in the user interface. Displaying the cue can includedisplaying the cue in the user interface corresponding to the voicecommand associated with the control, where the control is hidden fromthe user interface. Displaying the cue can include displaying the cue inthe user interface corresponding to the voice command associated withthe control, where the control is a global headset control. The cue canbe loaded from a control, the control indicating the cue and voicecommand.

In another embodiment, a system for displaying a user interface in aheadset computer can include a display module configured to provide auser interface in a headset computer. The display module can be furtherconfigured to, in response to user utterance of a cue toggle command,display at least one cue in the user interface. Each cue can correspondto a voice command associated with code to execute. The system canfurther include a command module configured to, in response to userutterance of the voice command, execute code associated with the voicecommand.

In another embedment, a method of developing a user interface in aheadset computer includes embedding a cue and a voice command in acontrol for the user interface. The method also includes providing thecontrol to the user interface, the user interface configured to displaythe cue responsive to a cue toggle command.

BRIEF DESCRIPTION OF DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIGS. 1A-1B are schematic illustrations of a headset computercooperating with a host computer (e.g., Smart Phone, laptop, etc.)according to principles of the present invention.

FIG. 2 is a block diagram of flow of data and control in the embodimentof FIGS. 1A-1B.

FIG. 3 is a diagram illustrating an example embodiment of a userinterface employed in the HSC.

FIG. 4 is a diagram illustrating an example embodiment of a userinterface after receiving a show commands voice command.

FIG. 5 is a flow diagram illustrating an example embodiment of a methodemployed by the present invention.

FIG. 6 illustrates a computer network or similar digital processingenvironment in which the present invention may be implemented.

FIG. 7 is a diagram of the internal structure of a computer (e.g.,client processor/device or server computers) in the computer system ofFIG. 6.

DETAILED DESCRIPTION OF THE INVENTION

FIGS. 1A and 1B show an example embodiment of a wireless computingheadset device 100 (also referred to herein as a headset computer (HSC))that incorporates a high-resolution (VGA or better) microdisplay element1010, and other features described below. The HSC 100 can include audioinput and/or output devices, including one or more microphones, inputand output speakers, geo-positional sensors (GPS), three to nine axisdegrees of freedom orientation sensors, atmospheric sensors, healthcondition sensors, digital compass, pressure sensors, environmentalsensors, energy sensors, acceleration sensors, position, attitude,motion, velocity and/or optical sensors, cameras (visible light,infrared, etc.), multiple wireless radios, auxiliary lighting,rangefinders, or the like and/or an array of sensors embedded and/orintegrated into the headset and/or attached to the device via one ormore peripheral ports (not shown in detail in FIG. 1B). Typicallylocated within the housing of headset computing device 100 are variouselectronic circuits including, a microcomputer (single or multicoreprocessors), one or more wired and/or wireless communicationsinterfaces, memory or storage devices, various sensors and a peripheralmount or mount, such as a “hot shoe.”

Example embodiments of the HSC 100 can receive user input throughsensing voice commands, head movements, 110, 111, 112 and hand gestures113, or any combination thereof. Microphone(s) operatively coupled orpreferably integrated into the HSC 100 can be used to capture speechcommands which are then digitized and processed using automatic speechrecognition techniques. Gyroscopes, accelerometers, and othermicro-electromechanical system sensors can be integrated into the HSC100 and used to track the user's head movement to provide user inputcommands. Cameras or other motion tracking sensors can be used tomonitor a user's hand gestures for user input commands. Such a userinterface overcomes the hands-dependent formats of other mobile devices.

The HSC 100 can be used in various ways. It can be used as a remotedisplay for streaming video signals received from a remote hostcomputing device 200 (shown in FIG. 1A). The host 200 may be, forexample, a notebook PC, smart phone, tablet device, or other computingdevice having less or greater computational complexity than the wirelesscomputing headset device 100, such as cloud-based network resources. Thehost may be further connected to other networks 210, such as theInternet. The headset computing device 100 and host 200 can wirelesslycommunicate via one or more wireless protocols, such as Bluetooth®,Wi-Fi, WiMAX, 4G LTE or other wireless radio link 150. (Bluetooth is aregistered trademark of Bluetooth Sig, Inc. of 5209 Lake WashingtonBoulevard, Kirkland, Wash. 98033.) In an example embodiment, the host200 may be further connected to other networks, such as through awireless connection to the Internet or other cloud-based networkresources, so that the host 200 can act as a wireless relay.Alternatively, some example embodiments of the HSC 100 can wirelesslyconnect to the Internet and cloud-based network resources without theuse of a host wireless relay.

FIG. 1B is a perspective view showing some details of an exampleembodiment of a headset computer 100. The example embodiment HSC 100generally includes, a frame 1000, strap 1002, rear housing 1004, speaker1006, cantilever, or alternatively referred to as an arm or boom 1008with a built-in microphone, and a micro-display subassembly 1010.

A head worn frame 1000 and strap 1002 are generally configured so that auser can wear the headset computer device 100 on the user's head. Ahousing 1004 is generally a low profile unit which houses theelectronics, such as the microprocessor, memory or other storage device,along with other associated circuitry. Speakers 1006 provide audiooutput to the user so that the user can hear information. Microdisplaysubassembly 1010 is used to render visual information to the user. It iscoupled to the arm 1008. The arm 1008 generally provides physicalsupport such that the microdisplay subassembly is able to be positionedwithin the user's field of view 300 (FIG. 1A), preferably in front ofthe eye of the user or within its peripheral vision preferably slightlybelow or above the eye. Arm 1008 also provides the electrical or opticalconnections between the microdisplay subassembly 1010 and the controlcircuitry housed within housing unit 1004.

According to aspects that will be explained in more detail below, theHSC display device 100 allows a user to select a field of view 300within a much larger area defined by a virtual display 400. The user cantypically control the position, extent (e.g., X-Y or 3D range), and/ormagnification of the field of view 300.

While what is shown in FIG. 1A is a monocular microdisplay presenting asingle fixed display element supported on the face of the user with acantilevered boom, it should be understood that other mechanicalconfigurations for the remote control display device 100 are possible.

FIG. 2 is a block diagram showing more detail of the HSC device 100,host 200 and the data that travels between them. The HSC device 100receives vocal input from the user via the microphone, hand movements orbody gestures via positional and orientation sensors, the camera oroptical sensor(s), and head movement inputs via the head trackingcircuitry such as 3 axis to 9 axis degrees of freedom orientationalsensing. These are translated by software in the HSC device 100 intokeyboard and/or mouse commands that are then sent over the Bluetooth orother wireless interface 150 to the host 200. The host 200 theninterprets these translated commands in accordance with its ownoperating system/application software to perform various functions.Among the commands is one to select a field of view 300 within thevirtual display 400 and return that selected screen data to the HSCdevice 100. Thus, it should be understood that a very large formatvirtual display area might be associated with application software or anoperating system running on the host 200. However, only a portion ofthat large virtual display area 400 within the field of view 300 isreturned to and actually displayed by the micro display 1010 of HSCdevice 100.

In one embodiment the HSC 100 may take the form of the HSC described ina co-pending US Patent Publication Number 2011/0187640 which is herebyincorporated by reference in its entirety.

In another embodiment, the invention relates to the concept of using aHead Mounted Display (HMD) 1010 in conjunction with an external ‘smart’device 200 (such as a smartphone or tablet) to provide information andcontrol to the user hands-free. The invention requires transmission ofsmall amounts of data, providing a more reliable data transfer methodrunning in real-time.

In this sense therefore, the amount of data to be transmitted over theconnection 150 is small-simple instructions on how to lay out a screen,which text to display, and other stylistic information such as drawingarrows, or the background colors, images to include, etc.

Additional data could be streamed over the same 150 or anotherconnection and displayed on screen 1010, such as a video stream ifrequired by the Host 200.

This invention relates to the viewing of context sensitive overlayswithin applications, on voice controlled HSCs 100.

The concept is the presentation of data, contextually, over a visual, ondemand. Overlays can be called up by the user with a voice command,typically “Show commands.” The voice command is standard across thesystem 100 and available at all times. This command causes HSC 100 todisplay applicable voice commands and other information in a contextsensitive and intuitive way.

The applicable commands are shown on a semi-transparent overlay of thecurrent screen view of display unit 1010. This allows the user to retainthe context of the screen he called the overlay up for.

The overlay and displayed applicable commands fade away after a shortperiod of time. This is accomplished by a timing mechanism thatrefreshes the screen view.

The applicable commands are displayed in order of relevance. The mostrelevant command is given more prominence in terms of placement overless relevant commands. 100 determines relevancy based on the currentcontext of the display 1010 contents.

Each screen in the relevant system is made up of user-interface (UI)components, some of which are ‘controls’. A control is a UI componentthat provides information to the user or enables some form offunctionality. Examples of controls are buttons, radio buttons, textboxes, check boxes, drop down menus, file menus, ribbon menus, livetiles, etc. Within the software developer's component library, these areavailable in their various forms, allowing customization of certainfeatures. For example, one such control might be a ‘button,’ simplyenabling the user to press it using a voice command available on thebutton. Controls, such as the ‘button’ control, are available to thedeveloper, for example, as part of the developer's component library orother library. The developer can insert the pre-coded control andcustomize it to his or her liking, instead of manually coding thecontrol from scratch.

A “Show Commands” function is built into the controls of the developerlibrary. When the developer, for example, creates a button and specifiesa text string to be written onto the button, the text string becomes thedefault voice command to activate the button, unless the developer (oruser) overrides the voice command. The control (e.g., the button) isconfigured to react to a “show commands” voice command by overlaying thetext string to activate the control over the control itself, or near thecontrol.

Every User Interface screen made available on the HSC 100 has theability to receive a “Show Commands” voice command (e.g., a systemcommand, available by default). Therefore, when a screen is constructedusing controls from the UI library, show commands functionality is builtin, providing guidance as to the voice commands available to the user.These voice commands are (by default) shown in context of the currentdisplayed contents (screen view).

Other available voice commands can also be placed within the showcommands overlay that are not associated with a visible control. Theseare placed in the overlay by the developer, adding a voice-command onlycontrol, or adding a hidden control, and provide a visual cue for voicecommands that are not associated with a button or other control.

FIG. 3 is a diagram 250 illustrating an example embodiment of a userinterface 202 employed in the HSC. The user interface 202, in thisembodiment, is an email application displaying a user's inbox. Eachemail listing is a respective email control 1-6 204 a-f. The user canopen each email by selecting each control. Each email control 204 a-f isprogrammed to be selected at least by a respective voice command.

FIG. 4 is a diagram 300 illustrating an example embodiment of a userinterface 302 after receiving a show commands voice command. The userinterface displays voice commands corresponding to each of the emailcontrols 204 a-f of FIG. 2. With respect to FIG. 3, voice commands 1-6304 a-4 correspond with email controls 1-6 204 a-f, respectively. Forexample, saying voice command 1 304 a (i.e., “Open E-mail 1”) causes theHSC to open the first email in the list.

The user interface 302 in show commands mode also shows a plurality ofimplicit voice commands 1-9 306 a-i. The implicit voice commands 1-9 306a-i do not correspond to any particular visual control of the userinterface, they are voice commands that are available to the user. Forexample, the user can say implicit voice commands 1 and 2 306 a-b tomove to the previous and next page, respectively. The user can draft anemail by saying implicit command 3 306 c. The user can manage his or heremail account by saying implicit command 4 306 d. The user can see hisor her accounts by saying implicit command 5 306 e. The user can switchfolders by saying implicit voice command 6 306 f The user can refreshthe inbox by saying implicit voice command 7 306 g.

Further, the user can go back to a previous screen by saying implicitvoice command 8 306 h. The user can return to a home screen by sayingimplicit voice command 9 306 i. Implicit voice commands 8 and 9 can beuniversal to all screens on the HSC. Voice commands 1-6 304 a-f andimplicit voice commands 1-7 306 a-g are local commands for thisparticular application. However, in other embodiments, implicit voicecommands 1-2 306 a-b can be global commands for moving to previous andnext pages of applications.

The voice command overlay aids the user by de-cluttering the screen ofoptions and buttons. The voice commands further help prompt the user tohow to use the system, which is especially useful while the user islearning how to use the device and voice commands.

FIG. 5 is a flow diagram 500 illustrating an example embodiment of amethod employed by the present invention. First, the method provides auser interface in a headset computer (502). Then, the method determineswhether it has received a cue toggle command, for example, over an audiochannel from a user utterance (504). If not, the method continueslistening for the cue toggle command (504). If so, however, the methodthen displays at least one cue in the user interface (506). Each cue isassociated with a corresponding voice command, which, when uttered,causes the system to execute code.

The system then determines whether it has received a voice command(e.g., a voice command shown by one of the cues) (508). If not, it keepslistening for a voice command (508). If so, however, it executes thecode associated with the voice command (510).

FIG. 6 illustrates a computer network or similar digital processingenvironment in which the present invention may be implemented.

Client computer(s)/devices 50 and server computer(s) 60 provideprocessing, storage, and input/output devices executing applicationprograms and the like. Client computer(s)/devices 50 can also be linkedthrough communications network 70 to other computing devices, includingother client devices/processes 50 and server computer(s) 60.Communications network 70 can be part of a remote access network, aglobal network (e.g., the Internet), a worldwide collection ofcomputers, Local area or Wide area networks, and gateways that currentlyuse respective protocols (TCP/IP, Bluetooth, etc.) to communicate withone another. Other electronic device/computer network architectures aresuitable.

FIG. 7 is a diagram of the internal structure of a computer (e.g.,client processor/device 50 or server computers 60) in the computersystem of FIG. 6. Each computer 50, 60 contains system bus 79, where abus is a set of hardware lines used for data transfer among thecomponents of a computer or processing system. Bus 79 is essentially ashared conduit that connects different elements of a computer system(e.g., processor, disk storage, memory, input/output ports, networkports, etc.) that enables the transfer of information between theelements. Attached to system bus 79 is I/O device interface 82 forconnecting various input and output devices (e.g., keyboard, mouse,displays, printers, speakers, etc.) to the computer 50, 60. Networkinterface 86 allows the computer to connect to various other devicesattached to a network (e.g., network 70 of FIG. 6). Memory 90 providesvolatile storage for computer software instructions 92 and data 94 usedto implement an embodiment of the present invention (e.g., contextsensitive overlays in a user interface code detailed above). Diskstorage 95 provides non-volatile storage for computer softwareinstructions 92 and data 94 used to implement an embodiment of thepresent invention. Central processor unit 84 is also attached to systembus 79 and provides for the execution of computer instructions.

In one embodiment, the processor routines 92 and data 94 are a computerprogram product (generally referenced 92), including a computer readablemedium (e.g., a removable storage medium such as one or more DVD-ROM's,CD-ROM's, diskettes, tapes, etc.) that provides at least a portion ofthe software instructions for the invention system. Computer programproduct 92 can be installed by any suitable software installationprocedure, as is well known in the art. In another embodiment, at leasta portion of the software instructions may also be downloaded over acable, communication and/or wireless connection. In other embodiments,the invention programs are a computer program propagated signal product107 embodied on a propagated signal on a propagation medium (e.g., aradio wave, an infrared wave, a laser wave, a sound wave, or anelectrical wave propagated over a global network such as the Internet,or other network(s)). Such carrier medium or signals provide at least aportion of the software instructions for the present inventionroutines/program 92.

In alternate embodiments, the propagated signal is an analog carrierwave or digital signal carried on the propagated medium. For example,the propagated signal may be a digitized signal propagated over a globalnetwork (e.g., the Internet), a telecommunications network, or othernetwork. In one embodiment, the propagated signal is a signal that istransmitted over the propagation medium over a period of time, such asthe instructions for a software application sent in packets over anetwork over a period of milliseconds, seconds, minutes, or longer. Inanother embodiment, the computer readable medium of computer programproduct 92 is a propagation medium that the computer system 50 mayreceive and read, such as by receiving the propagation medium andidentifying a propagated signal embodied in the propagation medium, asdescribed above for computer program propagated signal product.

Generally speaking, the term “carrier medium” or transient carrierencompasses the foregoing transient signals, propagated signals,propagated medium, storage medium and the like.

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

1. A method comprising: providing a user interface in a headsetcomputer, the user interface having a graphical user interface portion;in response to user utterance of a cue toggle command, displaying atleast one cue, each cue corresponding to a voice command associated withcode to execute, in the user interface; and in response to userutterance of the voice command, executing the code associated with thevoice command.
 2. The method of claim 1, further comprising: displayingthe interface without the cue at least one of prior to the cue togglecommand and after a subsequent cue toggle command.
 3. The method ofclaim 1, wherein displaying the cue includes displaying words thatactivate the voice command.
 4. The method of claim 1, wherein displayingthe cue includes displaying the cue in the user interface correspondingto the voice command associated with the control, the control displayedin the user interface.
 5. The method of claim 1, wherein displaying thecue includes displaying the cue in the user interface corresponding tothe voice command associated with the control, the control hidden fromthe user interface.
 6. The method of claim 1, wherein displaying the cueincludes displaying the cue in the user interface corresponding to thevoice command associated with the control, the control being a globalheadset control.
 7. The method of claim 1, wherein the cue is loadedfrom a control, the control indicating the cue and voice command.
 8. Asystem for displaying a user interface in a headset computer, the systemcomprising: a display module configured to provide a graphical userinterface portion to a user interface in a headset computer and furtherconfigured to, in response to user utterance of a cue toggle command,display at least one cue, each cue corresponding to a voice commandassociated with code to execute, in the user interface; and a commandmodule configured to, in response to user utterance of the voicecommand, execute code associated with the voice command.
 9. The systemof claim 8, wherein the display is further configured to display theinterface without the cue at least one of prior to the cue togglecommand and after a subsequent cue toggle command.
 10. The system ofclaim 8, wherein the display module is further configured to displaywords that activate the voice command.
 11. The system of claim 8,wherein the display module is further configured to display a cueincludes displaying a cue in the user interface corresponding to a voicecommand associated with the control, the control displayed in the userinterface.
 12. The system of claim 8, wherein the display module isfurther configured to display the cue in the user interfacecorresponding to the voice command associated with the control, thecontrol hidden from the user interface.
 13. The system of claim 8,wherein the display module is further configured to display the cue inthe user interface corresponding to the voice command associated withthe control, the control being a global headset control.
 14. The systemof claim 8, wherein the cue is loaded from a control, the controlindicating the cue and voice command.
 15. A method of developing a userinterface in a headset computer, the method comprising: embedding a cueand a voice command in a control for the user interface; and providingthe control to the user interface, the user interface configured todisplay the cue responsive to a cue toggle command.
 16. The method ofclaim 1, wherein the cue includes words that activate the voice command.17. The method of claim 1, wherein the cue corresponds to the voicecommand associated with the control displayed in the user interface. 18.The method of claim 1, wherein the cue corresponds to the voice commandassociated with the control hidden from the user interface.
 19. Themethod of claim 1, wherein the cue corresponds to the voice commandassociated with the control being a global headset control.
 20. Themethod of claim 1, wherein the cue is loaded from a control, the controlindicating the cue and voice command.